{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Amazon Augmented AI (Amazon A2I)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon Augmented AI (Amazon A2I) makes it easy to build the workflows required for human review of ML predictions. Amazon A2I brings human review to all developers, removing the undifferentiated heavy lifting associated with building human review systems or managing large numbers of human reviewers. \n",
    "\n",
    "You can create your own workflows for ML models built on Amazon SageMaker or any other tools. Using Amazon A2I, you can allow human reviewers to step in when a model is unable to make a high confidence prediction or to audit its predictions on an on-going basis. \n",
    "\n",
    "Learn more here: https://aws.amazon.com/augmented-ai/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Integrate human reviewers in Amazon Comprehend Text Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To incorporate Amazon A2I into your human review workflows, you need three resources:\n",
    "\n",
    "* **Worker Task Template** to create a Human Task UI for the worker. The worker UI displays your input data, such as documents or images, and instructions to workers. It also provides interactive tools that the worker uses to complete your tasks. For more information, see https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-instructions-overview.html\n",
    "\n",
    "* **Flow Definition** to create a Human Review Workflow.You use the flow definition to configure your human workforce and provide information about how to accomplish the human review task. You can create a flow definition in the Amazon Augmented AI console or with Amazon A2I APIs. To learn more about both of these options, see https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-create-flow-definition.html\n",
    "\n",
    "* **Human Loop** starts your human review workflow. When you use one of the built-in task types, the corresponding AWS service creates and starts a human loop on your behalf when the conditions specified in your flow definition are met or for each object if no conditions were specified. When a human loop is triggered, human review tasks are sent to the workers as specified in the flow definition.\n",
    "\n",
    "When using a custom task type, as this notebook will show, you start a human loop using the Amazon Augmented AI Runtime API. When you call StartHumanLoop in your custom application, a task is sent to human reviewers."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Install Latest SDK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, let's get the latest installations of our dependencies\n",
    "!pip install -qU pip\n",
    "!pip install -qU boto3\n",
    "!pip install -qU botocore"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "We need to set up the following data:\n",
    "* `region` - Region to call A2I\n",
    "* `bucket` - A S3 bucket accessible by the given role\n",
    "    * Used to store the sample images & output results\n",
    "    * Must be within the same region A2I is called from\n",
    "* `role` - The IAM role used as part of StartHumanLoop. By default, this notebook will use the execution role\n",
    "* `workteam` - Subsets or Group of people within your workorce to send the work to"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Role and Permissions\n",
    "\n",
    "The AWS IAM Role used to execute the notebook needs to have the following permissions:\n",
    "\n",
    "* ComprehendFullAccess\n",
    "* SagemakerFullAccess\n",
    "* S3 Read/Write Access to the BUCKET listed above\n",
    "* AmazonSageMakerMechanicalTurkAccess (if using MechanicalTurk as your Workforce)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setup Bucket and Paths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-1-835319576252/a2i-results\n"
     ]
    }
   ],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "\n",
    "sess   = sagemaker.Session()\n",
    "bucket = sess.default_bucket()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-1-835319576252/a2i-results\n"
     ]
    }
   ],
   "source": [
    "output_path = f's3://{bucket}/a2i-results'\n",
    "print(output_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Workteam or Workforce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A workforce is the group of workers that you have selected to label your dataset. You can choose either the Amazon Mechanical Turk workforce, a vendor-managed workforce, or you can create your own private workforce for human reviews. Whichever workforce type you choose, Amazon Augmented AI takes care of sending tasks to workers. \n",
    "\n",
    "When you use a private workforce, you also create work teams, a group of workers from your workforce that are assigned to Amazon Augmented AI human review tasks. You can have multiple work teams and can assign one or more work teams to each job."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create your Workteam, navigate here:  https://console.aws.amazon.com/sagemaker/groundtruth#/labeling-workforces/create\n",
    "\n",
    "\n",
    "After you have created your workteam, replace YOUR_WORKTEAM_ARN below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "WORKTEAM_ARN = 'arn:aws:sagemaker:us-east-1:835319576252:workteam/private-crowd/dsoaws'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visit: https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-permissions-security.html to add the necessary permissions to your role"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Client Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we are going to setup the rest of our clients. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "import json\n",
    "import uuid\n",
    "import time\n",
    "import boto3\n",
    "import botocore\n",
    "\n",
    "# Amazon SageMaker client\n",
    "sagemaker = boto3.client('sagemaker', region)\n",
    "\n",
    "# Amazon Comprehend client\n",
    "comprehend = boto3.client('comprehend', region)\n",
    "\n",
    "# Amazon Augment AI (A2I) client\n",
    "a2i = boto3.client('sagemaker-a2i-runtime')\n",
    "\n",
    "# Amazon S3 client\n",
    "s3 = boto3.client('s3', region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Control Plane Resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the Human Task UI using a Worker Task Template\n",
    "\n",
    "Create a human task UI resource, giving a UI template.  This template will be rendered to the human workers whenever human interaction is required.\n",
    "\n",
    "Below we've provided a simple demo template that is compatible with AWS Comprehend's Detect Sentiment API.\n",
    "\n",
    "For other pre-built UIs (70+), check: https://github.com/aws-samples/amazon-a2i-sample-task-uis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "template = r\"\"\"\n",
    "<script src=\"https://assets.crowd.aws/crowd-html-elements.js\"></script>\n",
    "\n",
    "<crowd-form>\n",
    "    <crowd-classifier\n",
    "      name=\"sentiment\"\n",
    "      categories=\"['Positive', 'Negative', 'Neutral', 'Mixed']\"\n",
    "      initial-value=\"{{ task.input.initialValue }}\"\n",
    "      header=\"What sentiment does this text convey?\"\n",
    "    >\n",
    "      <classification-target>\n",
    "        {{ task.input.taskObject }}\n",
    "      </classification-target>\n",
    "      \n",
    "      <full-instructions header=\"Text Classification Instructions\">\n",
    "        <p><strong>Positive</strong> sentiment include: joy, excitement, delight</p>\n",
    "        <p><strong>Negative</strong> sentiment include: anger, sarcasm, anxiety</p>\n",
    "        <p><strong>Neutral</strong>: neither positive or negative, such as stating a fact</p>\n",
    "        <p><strong>Mixed</strong>: when the sentiment is mixed</p>\n",
    "      </full-instructions>\n",
    "\n",
    "      <short-instructions>\n",
    "       Choose the primary sentiment that is expressed by the text.\n",
    "      </short-instructions>\n",
    "    </crowd-classifier>\n",
    "</crowd-form>\n",
    "\"\"\"\n",
    "\n",
    "def create_task_ui():\n",
    "    '''\n",
    "    Creates a Human Task UI resource.\n",
    "\n",
    "    Returns:\n",
    "    struct: HumanTaskUiArn\n",
    "    '''\n",
    "    response = sagemaker.create_human_task_ui(\n",
    "        HumanTaskUiName=taskUIName,\n",
    "        UiTemplate={'Content': template})\n",
    "    return response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arn:aws:sagemaker:us-east-1:835319576252:human-task-ui/ui-comprehend-a48f9914-0a95-4dd4-b4c7-bcb1d75bbb8a\n"
     ]
    }
   ],
   "source": [
    "# Task UI name - this value is unique per account and region. You can also provide your own value here.\n",
    "taskUIName = 'ui-comprehend-' + str(uuid.uuid4()) \n",
    "\n",
    "# Create task UI\n",
    "humanTaskUiResponse = create_task_ui()\n",
    "humanTaskUiArn = humanTaskUiResponse['HumanTaskUiArn']\n",
    "print(humanTaskUiArn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the Flow Definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we're going to create a flow definition definition. Flow Definitions allow us to specify:\n",
    "\n",
    "* The workforce that your tasks will be sent to.\n",
    "* The instructions that your workforce will receive. This is called a worker task template.\n",
    "* The configuration of your worker tasks, including the number of workers that receive a task and time limits to complete tasks.\n",
    "* Where your output data will be stored.\n",
    "\n",
    "This demo is going to use the API, but you can optionally create this workflow definition in the console as well. \n",
    "\n",
    "For more details and instructions, see: https://docs.aws.amazon.com/sagemaker/latest/dg/a2i-create-flow-definition.html."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Flow definition name - this value is unique per account and region. You can also provide your own value here.\n",
    "flowDefinitionName = 'fd-dsoaws-comprehend-' + str(uuid.uuid4()) \n",
    "\n",
    "create_workflow_definition_response = sagemaker.create_flow_definition(\n",
    "        FlowDefinitionName= flowDefinitionName,\n",
    "        RoleArn= role,\n",
    "        HumanLoopConfig= {\n",
    "            \"WorkteamArn\": WORKTEAM_ARN,\n",
    "            \"HumanTaskUiArn\": humanTaskUiArn,\n",
    "            \"TaskCount\": 1,\n",
    "            \"TaskDescription\": \"Identify the sentiment of the provided text\",\n",
    "            \"TaskTitle\": \"Classify Sentiment of Text\"\n",
    "        },\n",
    "        OutputConfig={\n",
    "            \"S3OutputPath\" : output_path\n",
    "        }\n",
    "    )\n",
    "flowDefinitionArn = create_workflow_definition_response['FlowDefinitionArn']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initializing\n",
      "Active\n",
      "Flow Definition is active\n"
     ]
    }
   ],
   "source": [
    "# Describe flow definition - status should be active\n",
    "for x in range(60):\n",
    "    describeFlowDefinitionResponse = sagemaker.describe_flow_definition(FlowDefinitionName=flowDefinitionName)\n",
    "    print(describeFlowDefinitionResponse['FlowDefinitionStatus'])\n",
    "    if (describeFlowDefinitionResponse['FlowDefinitionStatus'] == 'Active'):\n",
    "        print(\"Flow Definition is active\")\n",
    "        break\n",
    "    time.sleep(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Human Loops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detect Sentiment with AWS Comprehend"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have setup our Flow Definition, we are ready to call AWS Comprehend and start our human loops. In this tutorial, we are interested in starting a HumanLoop only if the SentimentScore returned by AWS Comprehend is less than 99%. \n",
    "\n",
    "So, with a bit of logic, we can check the response for each call to Detect Sentiment, and if the SentimentScore is less than 99%, we will kick off a HumanLoop to engage our workforce for a human review. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sample Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample_reviews = ['I enjoy this product', \n",
    "                  'I am unhappy with this product', \n",
    "                  'It is okay', \n",
    "                  'sometimes it works']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comprehend helper method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Will help us parse Detect Sentiment API responses \n",
    "def capsToCamel(all_caps_string):\n",
    "    if all_caps_string == 'POSITIVE':\n",
    "        return 'Positive'\n",
    "    elif all_caps_string == 'NEGATIVE': \n",
    "        return 'Negative'\n",
    "    elif all_caps_string == 'NEUTRAL':\n",
    "        return 'Neutral'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing sample_review: \"I enjoy this product\"\n",
      "SentimentScore of 0.9996746778488159 is above threshold of 0.99\n",
      "No human loop created. \n",
      "\n",
      "Processing sample_review: \"I am unhappy with this product\"\n",
      "SentimentScore of 0.9993742108345032 is above threshold of 0.99\n",
      "No human loop created. \n",
      "\n",
      "Processing sample_review: \"It is okay\"\n",
      "SentimentScore of 0.5957630276679993 is less than the threshold of 0.99\n",
      "Starting human loop with name: 2154327b-adf5-4012-a040-2ae4e926d19e  \n",
      "\n",
      "Processing sample_review: \"sometimes it works\"\n",
      "SentimentScore of 0.5166444182395935 is less than the threshold of 0.99\n",
      "Starting human loop with name: b5d87850-dd6d-4c2d-9174-62c69c509e71  \n",
      "\n"
     ]
    }
   ],
   "source": [
    "human_loops_started = []\n",
    "\n",
    "SENTIMENT_SCORE_THRESHOLD = 0.99\n",
    "\n",
    "for sample_review in sample_reviews:\n",
    "    # Call AWS Comprehend's Detect Sentiment API\n",
    "    response = comprehend.detect_sentiment(Text=sample_review, LanguageCode='en')\n",
    "    sentiment = response['Sentiment']\n",
    "    \n",
    "    print(f'Processing sample_review: \\\"{sample_review}\\\"')\n",
    "    \n",
    "    # Our condition for when we want to engage a human for review\n",
    "    if (response['SentimentScore'][capsToCamel(sentiment)]< SENTIMENT_SCORE_THRESHOLD):\n",
    "    \n",
    "        humanLoopName = str(uuid.uuid4())\n",
    "        inputContent = {\n",
    "            \"initialValue\": sentiment.title(),\n",
    "            \"taskObject\": sample_review\n",
    "        }\n",
    "        start_loop_response = a2i.start_human_loop(\n",
    "            HumanLoopName=humanLoopName,\n",
    "            FlowDefinitionArn=flowDefinitionArn,\n",
    "            HumanLoopInput={\n",
    "                \"InputContent\": json.dumps(inputContent)\n",
    "            }\n",
    "        )\n",
    "        human_loops_started.append(humanLoopName)\n",
    "        print(f'SentimentScore of {response[\"SentimentScore\"][capsToCamel(sentiment)]} is less than the threshold of {SENTIMENT_SCORE_THRESHOLD}')\n",
    "        print(f'Starting human loop with name: {humanLoopName}  \\n')\n",
    "    else:\n",
    "        print(f'SentimentScore of {response[\"SentimentScore\"][capsToCamel(sentiment)]} is above threshold of {SENTIMENT_SCORE_THRESHOLD}')\n",
    "        print('No human loop created. \\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Status of Human Loop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HumanLoop Name: 1de31c7a-9f2d-4ad5-9275-87830774bc96\n",
      "HumanLoop Status: InProgress\n",
      "HumanLoop Output Destination: {'OutputS3Uri': 's3://sagemaker-us-east-1-835319576252/a2i-results/fd-dsoaws-comprehend-7467ceeb-882c-4a4d-8221-867080061a3c/2020/05/18/19/15/06/1de31c7a-9f2d-4ad5-9275-87830774bc96/output.json'}\n",
      "\n",
      "\n",
      "HumanLoop Name: 2451d53b-581d-4e82-96ff-847dddabb63e\n",
      "HumanLoop Status: InProgress\n",
      "HumanLoop Output Destination: {'OutputS3Uri': 's3://sagemaker-us-east-1-835319576252/a2i-results/fd-dsoaws-comprehend-7467ceeb-882c-4a4d-8221-867080061a3c/2020/05/18/19/15/06/2451d53b-581d-4e82-96ff-847dddabb63e/output.json'}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "completed_human_loops = []\n",
    "for human_loop_name in human_loops_started:\n",
    "    resp = a2i.describe_human_loop(HumanLoopName=human_loop_name)\n",
    "    print(f'HumanLoop Name: {human_loop_name}')\n",
    "    print(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\n",
    "    print(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\n",
    "    print('\\n')\n",
    "    \n",
    "    if resp[\"HumanLoopStatus\"] == \"Completed\":\n",
    "        completed_human_loops.append(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wait For Human Workers to Complete Their Tasks\n",
    "Navigate to the link below and login with your email and password that you used when you setup the Private Workforce."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Navigate to the private worker portal and do the tasks. Make sure you've invited yourself to your workteam!\n",
      "https://bzxy1zzrn2.labeling.us-east-1.sagemaker.aws\n"
     ]
    }
   ],
   "source": [
    "workteamName = WORKTEAM_ARN[WORKTEAM_ARN.rfind('/') + 1:]\n",
    "print(\"Navigate to the private worker portal and do the tasks. Make sure you've invited yourself to your workteam!\")\n",
    "print('https://' + sagemaker.describe_workteam(WorkteamName=workteamName)['Workteam']['SubDomain'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check Status of Human Loop Again"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "HumanLoop Name: 1de31c7a-9f2d-4ad5-9275-87830774bc96\n",
      "HumanLoop Status: Completed\n",
      "HumanLoop Output Destination: {'OutputS3Uri': 's3://sagemaker-us-east-1-835319576252/a2i-results/fd-dsoaws-comprehend-7467ceeb-882c-4a4d-8221-867080061a3c/2020/05/18/19/15/06/1de31c7a-9f2d-4ad5-9275-87830774bc96/output.json'}\n",
      "\n",
      "\n",
      "HumanLoop Name: 2451d53b-581d-4e82-96ff-847dddabb63e\n",
      "HumanLoop Status: Completed\n",
      "HumanLoop Output Destination: {'OutputS3Uri': 's3://sagemaker-us-east-1-835319576252/a2i-results/fd-dsoaws-comprehend-7467ceeb-882c-4a4d-8221-867080061a3c/2020/05/18/19/15/06/2451d53b-581d-4e82-96ff-847dddabb63e/output.json'}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "completed_human_loops = []\n",
    "for human_loop_name in human_loops_started:\n",
    "    resp = a2i.describe_human_loop(HumanLoopName=human_loop_name)\n",
    "    print(f'HumanLoop Name: {human_loop_name}')\n",
    "    print(f'HumanLoop Status: {resp[\"HumanLoopStatus\"]}')\n",
    "    print(f'HumanLoop Output Destination: {resp[\"HumanLoopOutput\"]}')\n",
    "    print('\\n')\n",
    "    \n",
    "    if resp[\"HumanLoopStatus\"] == \"Completed\":\n",
    "        completed_human_loops.append(resp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### View Task Results  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once work is completed, Amazon A2I stores results in your S3 bucket and sends a Cloudwatch event. Your results should be available in the S3 OUTPUT_PATH when all work is completed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{   'flowDefinitionArn': 'arn:aws:sagemaker:us-east-1:835319576252:flow-definition/fd-dsoaws-comprehend-7467ceeb-882c-4a4d-8221-867080061a3c',\n",
      "    'humanAnswers': [   {   'answerContent': {   'sentiment': {   'label': 'Neutral'}},\n",
      "                            'submissionTime': '2020-05-18T19:19:07.688Z',\n",
      "                            'workerId': '31199406719ea510'}],\n",
      "    'humanLoopName': '1de31c7a-9f2d-4ad5-9275-87830774bc96',\n",
      "    'inputContent': {'initialValue': 'Neutral', 'taskObject': 'It is okay'}}\n",
      "\n",
      "\n",
      "{   'flowDefinitionArn': 'arn:aws:sagemaker:us-east-1:835319576252:flow-definition/fd-dsoaws-comprehend-7467ceeb-882c-4a4d-8221-867080061a3c',\n",
      "    'humanAnswers': [   {   'answerContent': {'sentiment': {'label': 'Mixed'}},\n",
      "                            'submissionTime': '2020-05-18T19:19:03.253Z',\n",
      "                            'workerId': '31199406719ea510'}],\n",
      "    'humanLoopName': '2451d53b-581d-4e82-96ff-847dddabb63e',\n",
      "    'inputContent': {   'initialValue': 'Positive',\n",
      "                        'taskObject': 'sometimes it works'}}\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "import pprint\n",
    "\n",
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "\n",
    "for resp in completed_human_loops:\n",
    "    splitted_string = re.split('s3://' +  bucket + '/', resp['HumanLoopOutput']['OutputS3Uri'])\n",
    "    output_bucket_key = splitted_string[1]\n",
    "\n",
    "    response = s3.get_object(Bucket=bucket, Key=output_bucket_key)\n",
    "    content = response[\"Body\"].read()\n",
    "    json_output = json.loads(content)\n",
    "    pp.pprint(json_output)\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The End!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
